Video-editing workflow methods and apparatus thereof

ABSTRACT

Video-editing workflow methods and apparatus thereof are disclosed. A user drops off (or send) to a merchant that provides video-processing services the analog or digital videotape. The videotape can be processed (digitized if necessary) into both a high-resolution and low-resolution video stream. The low-resolution video stream could be provided to a customer either in the form of a CD/DVD or accessible for download from the Internet. While at home on a personal computer or consumer video edition device, the user could perform the edits and cuts on the low-resolution video stream. The edits to the video stream, which are very small, can then be sent into the video processing service and re-rendered on the high-resolution video stream.

BACKGROUND OF THE INVENTION

[0001] 1. Field of Invention

[0002] The invention relates generally to digital image processingsystems. Video-editing workflow methods and apparatus thereof aredescribed.

[0003] 2. Description of Relevant Art

[0004] When a user performs image operations using digital imageprocessing programs such as Adobe Photoshop™ or PhotoDeluxe™, imageoperations are performed directly on the raw pixels of the image. Sincemost imaging applications only perform image operations on oneresolution, usually the highest resolution, these operations aresometimes very slow, even on the fastest computers. If an applicationcould work at a lower resolution for display purposes, the processingtime would significantly decrease, thus increasing the productivity ofthe user. While it is sometimes possible for an application to work onlower-resolution image data, when the image with all the applied imageoperations is to be saved, the full-resolution image data must beprocessed at that time. If this step were not performed, the saved imagewould only contain low-resolution image data. While this is one option,this is not desired since it would not be possible to obtain an imagerasterized at a higher resolution.

[0005] In general, an application is forced to generate one resolution(usually the highest), even though lower-resolution data could be usedsince the output may be targeted to a low-resolution display device andnot a high-resolution printer. Unfortunately, it is not possible for theapplication to know apriori how the image will be used in the future,thus a high-resolution image is usually created.

[0006] A more desirable solution would be if a low-resolution (i.e.resultant) image could be generated based on the sequence of imageoperations, but with “knowledge” about the original high-resolutionimage (i.e. the digital negative) as well as the list of imageoperations (i.e. the edit list). The digital negative is the user'soriginal, unmodified image stored at the highest resolution possible.The edit list is a sequence of image operations or image transforms thatare to be applied to an image. The resultant image is the result ofapplying the specified edit list applied to the digital negative at somespecified resolution, which includes a reference (i.e. internal orexternal link) to the digital negative along with the optional editlist.

[0007] There have been recent developments in technology that haveattempted to solve this problem. An imaging application that supportsFlashPix™ technology, developed by a consortium of companies comprisingof the Digital Imaging Group (DIG), can display and manipulate on-screenimages at a lower-resolution, but retains access to thehigher-resolution image data. The FlashPix™ file contains the followingitems: (a) the original higher-resolution image data, (b) a very limitedset of image transforms (such as, rotation, translation, cropping, colortwist, blur/sharpen adjustment, brightness/contrast adjustment), and (c)an optional resultant image. While this appears to resolve the problemslisted above, there are several limitations. The most important of whichis that the original image data, the image transforms, and the optionalresultant image are stored together in one central file. This does notprovide for a distributed imaging architecture as describe by theinvention, nor is it a lightweight solution needed for transmission ofimages in a low-bandwidth network environment. Additional limitations ofFlashPix are that the file format is based on a proprietary, complexformat referred to as COM/Structured Storage which, in the current formused by FlashPix™, does not support the ideas put forth by thisinvention. Further, the image transforms supported are limited to a fewoperations. The solution described by this invention provides a muchmore flexible model.

[0008] Other attempts have been made that try to remedy the problemoutlined above, but with limited success. The Internet Imaging Protocol(IIP) has been developed that provides for optimized access and displayof a FlashPix files over the network. When using IIP, a FlashPix filethat resides on a serve can be efficiently rendered. A client (i.e. Webbrowser) must contain executable code (such as a Java applet, ActiveXcontrol, or browser plug-in) that allows it to request portions of theFlashPix file. This involves downloading of fully rendered tiles orportions of the image at different resolutions. While this does providefor efficient display of a FlashPix file over a low bandwidth networkconnection, it does not solve the generate problem of linking thelow-resolution resultant image file back to the high-resolution digitalnegative and edit list. IIP is simply a method for serving up andviewing large FlashPix files over the network in an efficient manner.

[0009] Other variants of FlashPix exist, including LivePicture's FITStechnology. The FITS format allows the image operations to be performedon low-resolution image data and saved, thus eliminating the need to doa time consuming render of the high-resolution image data when savingthe file. The re-rendering at high resolution is done once the outputresolution is determined. However, it does not solve the general problemof linking the low-resolution resultant image file back to thehigh-resolution digital negative and edit list.

[0010] In addition to providing general improvements for distribution ofimages, there are several web-based imaging solutions that are greatlyenhanced by incorporation of this invention, but up until now could notprovide similar novel functionality.

[0011] A variety of companies offer creation and distribution of on-linegreeting cards (Blue Mountain Arts, Yahoo, etc.) and calendars. Somealso offer the ability to customize cards and calendars by compositing apersonal photo with their content. In general, these are offered as afree service and utilize low-resolution content and/or user's photos.Even if a high-resolution photograph were uploaded, it might bedownsampled and composited with the greeting card artwork. Thiscomposition, in low-resolution form, is then sent, possibly throughe-mail, to the recipient of the card or calendar.

[0012] This becomes even more important as higher-quality on-linecontent providers (i.e. Corbis, Getty, etc.) enter into this space.These companies can offer much higher quality content, but expect to becompensated if this content is taken advantage of. One model mightinvolve users creating greeting cards or calendars, optionallycomposited with their photos, but only low-resolution resultant output(and optionally with a watermark) to be generated and distributed forfree.

[0013] Peer-to-peer Web-based file sharing is a model that is becomingincreasingly popular. Several applications now exist today, includingGnutella (OpenSource application for peer-to-peer file sharing) andNapster (peer-to-peer audio sharing). A same opportunity exists fordirect peer-to-peer sharing of image files. As with any of theseapplications, and more problematic for images, is that the size of thefiles could result in an almost unusable user experience when image datais transferred over low-bandwidth network connections. Since purepeer-to-peer (no central intermediate image cache) involves directconnection between two or more end-user's computing devices, therequirement of transferring a large amount of image data, even on ahigh-bandwidth connection, almost makes this idea infeasible.

[0014] For photos today shared using direct peer-to-peer connections,the option is either send the high-resolution image that results in slowtransmission or a low-resolution image that is faster, but poorerquality. There is no direct way to send a low resolution image, but withan automatic method of accessing the high-resolution image data ifneeded.

[0015] Embedded imaging devices can also take advantage of thisinvention to provide increased functionality and performance notpossible today. Today certain digital cameras, including the Kodak DC265and DC290, provide in-camera image processing software support. Withthis software, a user can apply specific effects or other imageprocessing operations to an image after it has been captured. Althoughonly a small thumbnail of the image can be viewed on the LCD screen, thefull resolution image must be processed and stored in the camera memory(such as a CompactFlash card). While it might be possible to perform anoptimization step that allows the low-resolution thumbnail to begenerated so it can be quickly displayed on the LCD screen, thefull-resolution image must still be processed and stored in the cameramemory. Even worse, if the original image is also to be retained, twocopies of the high-resolution image must be stored (the original and theone with the image operation applied).

[0016] In the realm of video image processing. Adobe Premiere andAfterEffects are products that provide powerful real-time video editing.For these and related products, users edit low-resolution (lower framerate or less image resolution) video sequences to makes cuts, addingtransitions and special effects, as well as other video editingoperations. The commands are captured and then executed on thefull-resolution video stream into a desired video format (such as Quicktime, NTSC, or PAL) at a specified target resolution. While this doesprovide for on-demand rasterization based on the target output device,it is done so in a centralized manner. Inside an Adobe Premiere orAfterEffects project file, all the information and links to thenecessary original (high-resolution) video clips are required and mustbe accessible.

[0017] While this approach is standard in the field of video editing, itrequires a massive amount of storage space and processing power torender full-resolution, high-quality video output. Even with a high-endsystem, it still could take a long time (10 minutes to several hours)too fully render the complete video clip. This solution is not viablefor a consumer-based video-editing device.

[0018] What is desired is an alternative approach for creating avideo-editing workflow that requires much less processing power forend-users.

SUMMARY OF THE INVENTION

[0019] The invention relates to improved video-editing workflow methodsand apparatus thereof are described.

[0020] In one embodiment, a user drops off (or send) to a merchant thatprovides video-processing services the analog or digital videotape. Thevideotape can be processed (digitized if necessary) into both ahigh-resolution and low-resolution video stream. The low-resolutionvideo stream could be provided to a customer either in the form of aCD/DVD or accessible for download from the Internet. While at home on apersonal computer or consumer video editing device, the user couldperform the edits and cuts on the low-resolution video stream. The editsto the video stream, which are very small, can then be sent into thevideo processing service and re-rendered on the high-resolution videostream. In this case, the resultant image might not contain any videoimage at all, but contains the “knowledge” of how to render thehigh-resolution video stream.

[0021] In particular, resultant image contains a reference back to thehigh-resolution video stream and the edit list of image operations thatwere applied to the video stream. Although the processing time to fullyrender the high-resolution video stream may still take a long time, itcan be performed as a batch operation off-line. Once complete, the fullyedited high-resolution video stream can be given to the end user in somestandard medium (such as CD, DVD, or tape).

[0022] These and other advantages of the present invention will becomeapparent upon reading the following detailed descriptions and studyingthe various figures of the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The invention, together with further advantages thereof, may bestbe understood by reference to the following description taken inconjunction with the accompanying drawings in which:

[0024]FIG. 1 shows a block diagram of a digital image processor systemin accordance with an embodiment of the invention is shown.

[0025]FIG. 2 illustrates a particular implementation of the imageprocessing engine shown in FIG. 1.

[0026]FIGS. 3A and 3B are examples of resultant proxy image files andassociated edit list files in accordance with an embodiment of theinvention.

[0027]FIG. 4 shows a distributed system in accordance with an embodimentof the invention.

[0028]FIG. 5 shows a digital camera system in accordance with anembodiment of the invention.

[0029]FIG. 6 shows a flowchart is shown detailing a process carried outby the digital image processing engine shown in FIG. 1.

[0030]FIG. 7 illustrates a flowchart detailing a process whereby a usercreates a intelligent “e-card” in accordance with an embodiment of theinvention.

[0031]FIG. 8 illustrates a process for facilitating efficient videoediting, in a distributed environment in accordance with an embodimentof the invention.

[0032]FIG. 9 illustrates a computer system or any imaging-enabledcomputing appliance/device that employed to implement the invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] Broadly speaking, the invention relates a novel approach andmechanism that allows a low-resolution digital greeting card image ordigital calendar image to contain a link to an associated originaldigital negative and edit list. Unlike conventional approaches to imageediting, the distributed nature of the invention provides substantiallyimproved flexibility. More specifically, the entire high-resolutionimage must not always be rasterized or distributed, since through thisinvention, it is possible to easily obtain access to the digitalnegative and edit-list of operations, thus allow additional resolutionsto be generated on-demand as needed by an application or output device.Furthermore, in a distributed system utilizing, for example, the Web, CDmedia, or directly from a digital camera or other imaging appliancedevice, the invention provides that a user's digital negative can resideanywhere within this distributed system. In this case, a low-resolutionimage object can be distributed that will facilitate very fast downloadand display since it may be of a much smaller size compared to theoriginal high-resolution image.

[0034] The image object may contain a proxy raster image, or resultantimage, of its contents in any industry standard format (i.e. JPEG, TIFF,PNG, etc) provided on-demand rasterization is not available. The proxyraster image can be a fully rendered image of the digital negative withthe image operations specified by the edit list applied at someresolution. This allows for very fast download and immediately displayof the image, since it most likely will be of a lower resolution withpossibly a higher compression ratio. In the described embodiment, alow-resolution (i.e. resultant) image could be generated based on thesequence of image operations, but with “knowledge” about the originalhigh-resolution image (i.e. the digital negative) as well as the list ofimage operations (i.e. the edit list). The digital negative is theuser's original, unmodified image stored at the highest resolutionpossible. The edit list is a sequence of image operations or imagetransforms that are to be applied to an image. The resultant image isthe result of applying the specified edit list applied to the digitalnegative at some specified resolution, which includes a reference (i.e.internal or external link) to the digital negative along with theoptional edit list.

[0035] Further, in a preferred embodiment, it will be in an industrystandard format, thus allowing it to be displayed by most applicationsand Web browsers. More importantly, if the digital negative or edit listassociated with the proxy image are not available, the user will stillhave a preview of what the image composition should look like. Thisallows a user to view the proxy image (or image composition) at a givenresolution, using a standard application, but retains its ability to berasterized at a higher resolution at a later time, based on specificapplication or device requirements. It is important to note that thisinvention works within the framework of existing industry standards,thus allowing any application to view the proxy image if the applicationdoes not have support for the architecture outlined by this invention.

[0036] It should be noted, when an application displays the resultantproxy image, either because the original image cannot be re-resterizedor the application requires immediate display of the resultant proxyimage, the proxy image might not actually match the resolution of theoutput device. If this occurs, the resultant proxy image can easily beresampled to match the resolution of the output device, a process wellknown in the art.

[0037] This is particularly advantageous when viewing images over anetwork. If two end-user computers are configured in a peer-to-peerconnected environment, such that sharing of photos and other multi-mediaobjects are permitted, this invention will create a much more desirableexperience, since only low-resolution photos need to be transmitted. Itis only when a user needs access to higher resolution image data, suchas when printing or zooming into an image, does the higher resolutiondata need to be optionally accessed. Even in a situation where a user isviewing photos from a central photos sharing service on a network, suchas Zing.com or Kodak's PhotoNet, this invention will permit users quickaccess to the lower-resolution image data, but retains the ability toaccess higher resolution image data.

[0038] It should be noted that an additional advantage of the inventionis that the resultant proxy image can be very small and dependent upon aspecific device. For example, the resultant proxy image derived from aset of edit operations applied by a digital camera having a small LCDdisplay may in fact be a low-resolution thumbnail image. In this way,the resultant proxy image will be properly displayed and optimized forthe LCD display on the camera. By using a linked edit list, the editingoperations can be applied on the full resolution version of the originalhigh-resolution image (i.e., the digital negative) at a later timewhenever a higher-resolution resultant image is needed. Clearly then,two of the benefits from this approach are that the image processingoperations are very quick since only those pixels identified by theediting operations at a given resolution are processed resulting in asmaller resultant proxy image thereby preserving in-camera memorystorage space.

[0039] Based on the editing operation, the software application mayrequire a much smaller number of pixels from the digital negative if theediting operation can be performed in a resolution-independent manner.This provides for more efficient CPU utilization for processing ofoperations. Additionally, when the actual image data of the digitalnegative is requested in a distributed environment, higher networkutilization is possible since lower-resolution image data can betransmitted between two computing devices provided the output device islower resolution than the original digital negative.

[0040] However, for cases in which the editing operation is notresolution-independent, the software performing the operation may chooseto access more pixels from the digital negative for the editingoperation, but resample (reduce the number of pixel in) the output imageafter the editing operations have been performed, thus resulting insmaller resultant proxy image.

[0041] In general, the “edit list” is normally associated with aresultant image, such that the resultant image can be recreated at adesired resolution as needed. This “edit list” contains all thenecessary information about how to perform this reconstruction. It willinclude the reference(s) to the digital negative(s) and any additionalmultimedia assets needed for this reconstruction. It should be notedthat several multimedia assets (including several digital negatives suchas still images, video images, vector artwork, test, etc.) might berequired for this reconstruction. This is because the resultant imagemay be a composite of more than one multimedia asset.

[0042] It will also contain an optional list of editing operations thatmust be performed. The “edit list” can be considered both the “glue”that identifies all digital negatives and other multimedia assets thatare needed to reconstruct the resultant image and the “script” on howthe digital negatives and other multimedia assets are rendered such thatthe resultant image can be recreated, at the current resolution of theresultant image or at a different resolution.

[0043] It should be realized that an “edit list ” can take the form of avariety of different embodiments, three of which are described below.This is meant to describe different variants of an edit list, but not anall-inclusive list. The first embodiment is a list that describes a setof individual operations that are to be performed on a given photo.These edit operations can be applied to a photo to generate theresultant image at the desired output resolution. The advantage of thisstructure is that new edits can be added to the end of the list orremoved from the end of the list through an “undo” operation. A firstsample piece of XML code is included for illustrative purposes:

[0044] <!-- Edit list example that consists of a list of operationsapplied -->

[0045] <!-- This example demonstrates when the edit list is a list ofsequential operations -->

[0046] <!-- that is applied to the digital negative. In the examplebelow, an operation can -->

[0047] <!-- be applied more than once, and in any order. This followsthe conventional -->

[0048] <!-model of a list of operations as entered by the user in whichthe list can be -->

[0049] <!-at a later time undone if desired (via an undo operation). -->

[0050] <PICTURE-IQ-EDIT-LIST>

[0051] <DIGITAL-NEGATIVE-LINK type=“highest_resolution” height=“1600”width=“1200” href=“www.pictureiq.com/samples/nicholasjpg”content-type=“image/jpeg”/>

[0052] <EDIT-SCRIPT type=“sequential_list_of operations”>

[0053] <!-- auto_fix is an operation that is applied to the digitalnegative, and does -->

[0054] <!-- not take any parameters -->

[0055] <EDIT-COMMAND type=“auto_fix”/>

[0056] <!-- color_adjustment is applied to the digital negative, withthe following parameters -->

[0057] <EDIT-COMMAND type=“color_adjustment” red=“10” green=“−20”blue=“50”/>

[0058] <!-- next a warp is applied with the following parameter -->

[0059] <EDIT-COMMAND type=“warp” amount=“20”/>

[0060] <!-- finally, another color_adjustment command is specified -->

[0061] <EDIT-COMMAND type=“color_adjustment” red=“−10” green=“00”blue=“50”/>

[0062] </EDIT-SCRIPT>

[0063] An alternative embodiment is to use this list to represent astate description of each operation applied to the photo, in whichcertain operations can be turned on or off. Each operation can stillhave an optional parameter list associated with the command, but thereis a single “state” for the operation. It is either “on” or “off”, andif “on”, it can have a parameter associated with it. The advantage ofthis structure is that the operations and the interaction between theoperations are clearly defined by the structure, thus providingconsistent results regardless of the order that the user has applied theoperations. Either the XML syntax could define the order in which theoperations are performed, similar to the first embodiment, or theapplication can define it, independent of the order in which they appearin the XML file. The key point is that each operation is either “on” or“off” and would normally not appear more than once on the edit list. Asecond sample piece of XML code is included for illustrative purposes:

[0064] <!-- Edit list example that consists of a state description ofavailable -->

[0065] <!-- operations each operation is either “on” or “off”, with itsspecified

[0066] <!-- parameter list each operation should only be specified onceand the -->

[0067] <!-- order in which it is applied can be strictly specified bythe application -->

[0068] <PICTURE-IQ-EDIT-LIST>

[0069] <DIGITAL-NEGATIVE-LINK type=“highest_resolution” height=“1600”width=“1200” href=“www.pictureiq.com/samples/nicholasjpg”content-type=“image/jpeg” />

[0070] <EDIT-SCRIPT type=“state description_of_operations”>

[0071] <!-- warp is specified, with its parameter, but is currently notenabled -->

[0072] <!-- (active). If the user enabled warp again, the applicationcould -->

[0073] <!-- use either a preset value or the one specified in this file(20), -->

[0074] <!-- as required by the application. -->

[0075] <EDIT-COMMAND state=“off” type=“warp” amount=“20”/>

[0076] <!-- color_adjustment and auto_fix are both enabled and appliedto the

[0077] <!-- digital negative. -->

[0078] <EDIT-COMMAND state=“on” type=“color_adjustment” red=“10”green=“−20” blue=“50”/>

[0079] <EDIT-COMMAND state=“on” type=“auto_fix”/>

[0080] </EDIT-SCRIPT>

[0081] </PICTURE-IQ-EDIT-LIST>

[0082] A third embodiment combines the benefits of the first twoembodiments. The edit list not only represents the state description ofoperations, but also includes the ability for one or more of theoperations that are part of the state description to be a list ofindividual operations that are to be applied to the photo, thus mergingthe two methods

[0083] It should be noted, this architecture also allows an applicationto describe access to multiple resolutions of a digital negative from asingle file. For example, multiple resolutions of the same image (atdifferent resolution) could be externally linked via the “edit list”that describes the cross-reference between the various distributed imagefiles. In this case, the “edit list” is the “glue” that holds thestructure indicative of how each file is interrelated, but notnecessarily the “script” that describes additional editing operations,here the “script” portion of the edit list is empty or null. Forexample, in one implementation, the edit list includes a pointer thatpoints to a remote image file having a higher (or lower) resolution thanthe digital negative or, in some cases, a resultant proxy image.

[0084] The capability for linking an edit list(s) from a low-resolutionproxy resultant image provides substantial advantages for e-commerceapplications. For example, photo content providers can distributelower-resolution images for free via the Web or a CD that containreferences back to the original high-resolution digital negative (s)that can reside either on a Web site, CD media, or other removablemedia. Photo Web sharing sites can also use this invention to enableusers to link photos, or photo greeting cards (and other compositionssuch as multi-page albums) to the high-resolution image data and anoptional “script” within the edit list for rendering of the photo (orcard or album) at a higher-resolution.

[0085] An added security feature could also be provided that allows thehigh-resolution image to be locked with, for example, an encryption keyto prevent unauthorized access unless the customer has actuallypurchased the rights to the given photo. It is a benefit of thisinvention that the “key” for how the high-resolution digital photo canbe accessed is specified by the edit list (either embedded or linked) ofthe low-resolution proxy image. In addition, it has a further advantagethat this meta-data could also include ordering information such asprice, restrictions and the like.

[0086] It should be noted that an “edit list” of a multimedia asset,such as a digital image, can take many forms, such as for example, asingle file representative of one digital negative and one separate XMLfile associated with the edit list for a given resultant image. In othercases, the edit list can be part of a file, referred to as a catalogfile, that includes several different “edit lists” each being associatedwith a different multimedia asset. In one implementation, a catalog filecan include several digital negatives (and their corresponding resultantimages) and a single XML file that contains several different XML editlists, one for each resultant image. In this way, the catalog file is anefficient approach for maintaining edit lists for a number of images ina single file.

[0087] The proxy image may also be obscured or “stamped” to render itunsuitable for printing or distribution, while still retaining thecapability to access the original image if the user purchases suchrights. The proxy image could also contain either a visible or invisiblewatermark, thus providing an additional form of identification andprotection to the image.

[0088] It should be noted that both the digital negative and theresultant proxy image can be any industry-standard image formats (i.e.JPEG, PNG, GIF, etc) or even a proprietary file format (although not inthe preferred embodiment) since the edit list and links between theimages can be encoded in private meta-data embedded inside or externallylinked to the image files. In this way, any application regardlesswhether or not it is cognizant of the invention is able to open and viewany image that contain the private meta-data, since the resultant imagecontains a rasterized version of the image, albeit possibly a lowerrendered resolution. Of course, however, applications that do recognizeand support the invention will have the ability to re-render the image,as necessary, to achieve any desired result.

[0089] The invention provides for the ability to efficiently downloadlower-resolution images from the Web, but preserves the ability to berasterized at a higher resolution either locally or remotely at a latertime when needed, thus preserving the bandwidth.

[0090] In one embodiment, it allows a user to quickly download alow-resolution resultant image in an industry standard format (such asJPEG, PNG, TIFF, or others), but maintains the knowledge about how ahigher resolution image can be created. In this case, the low-resolutionresultant image has associated with it an edit-list of operations thatmust be applied to the digital negative. This allows for quick viewing,but with the option to obtain higher-resolution image data as needed.

[0091] It should be noted that this invention can be used to enhance thetransmission of image data, no matter what the environment or transportmedium. For example, downloading of user's images from a central Website (such as Zing.com, PhotoNet. and other on-line photo communities iproviders). This invention can also facilitate much improved transfer ofimages over a network in which two end user's computers are connected ina direct peer-to-peer fashion. Further, this invention can be usedequally well, when transmitting images over a wireless networkconnection.

[0092] In another embodiment, a user can download a low-resolution“Intelligent e-Card” from a Web site represented as an industry standardJPEG that contains the rasterized composition of photos, cards, text,and any other multimedia assets. While the JPEG proxy image may beadequate for on-screen viewing or a preview print, the proxy image isembedded with knowledge about how the high-resolution artwork and/orphotos can be accessed and composited, providing the application withthe necessary information for the rasterization at a higher-resolution.The benefit is that a very small sized proxy image (such as an industryJPEG, PNG, TIFF, or other format) can be initially downloaded for reviewvery quickly and at a later time, higher-resolution data can beaccessed.

[0093] Assuming that a higher-resolution “e-card” is to be created, thisinvention permits the creation of the high-resolution data eitherlocally on the client computing device or on a remote computing device(such as a server computer). In the case that rendering occurs on theclient computing device, higher-resolution image data is obtained fromthe digital negative (s) and the edit lists are processed generating ahigher resolution “e-card”. Alternatively, the client computing devicecould request that the rendering be performed remotely (such as on ahigh-performance server), but where the rendered higher resolution“e-card” be subsequently transferred to the client computer in its finalform. It should be noted that this architecture could be configured tomeet the requirements of many different e-commerce sites. Thelow-resolution “Intelligent e-Card” could be watermarked, thusencouraging the user to purchase a higher-resolution representation.Further, since the low-resolution resultant image has the knowledgeabout how to recreate a higher-resolution version, the e-commerce sitecould require, when this high-resolution data is accessed, that the userpurchased the rights to access and use the high-resolution data.

[0094] It should also be noted, this is not limited to greeting cards,but could also be applied to the creation and distribution of albums,calendars, and virtually any other creation that involves incorporationof photographs or other image data into a composition.

[0095] It should also be noted that in all these embodiment, severalperformance optimizations can easily be attained. In particular, when aphoto or composition (such as a card, calendar, or album) is printedthrough a remote location, the customer (end user) may not even need totransmit much data from their local computing device. Although thecustomer may only have direct (local) access to a low-resolution proxyimage, the resolution that best matches the output device (such as ahigh-resolution printer) will be rasterized based on the originaldigital negative and the edit list stored with the proxy image. Moreimportantly, if the high-resolution image resides on the Web (such asZing.com or PhotoNet) and output is going to a remote device (such as aprinter at Kinkos), very little client bandwidth is required from thelocal user's computing device. Since the resultant image has associatedwith it the edit list and references to the digital negative(s) data,only that small amount of information must be transferred from theuser's local computing device to the remote device (such as a printservice).

[0096] In a particular video-based embodiment described below, the useris able to download and edit a low-resolution digital video stream fromthe Web. The video editing commands can be passed to the server forprocessing on the full-resolution video stream. The commands areassociated with the low-resolution digital-video stream that maintains areference back to the high-resolution video stream on a server. At alater point in time, the editing commands can be processed on thehigh-resolution video steam remotely. This requires only a minimalamount of data to be transmitted (only the editing commands), resultingin less bandwidth and processor requirements on the client side.

[0097] In this way, the invention provides a very powerful distributedmulti-resolution, and non-destructive imaging model. It is alsoplatform, operating system, and device independent. Furthermore, it isextremely lightweight and compatible with many other standards. Thereare many unique aspects to this invention that will be realized afterfurther review. The distributed characteristic of this invention is theunique aspect that provides the real flexibility and power not found inany application or system today. While this invention was originallydeveloped for a digital imaging (still and video) consumer electronicsdevices, its applicability spans any computing device that performsoperations on multimedia content.

[0098] Referring now to FIG. 1, a block diagram of a digital imageprocessor system 200 in accordance with an embodiment of the inventionis shown. Digital image processor system 200 includes an image source202 arranged to provide any number of digital or analog image inputsignals for processing. The image source 202 can provide a digital imagestream that can take the form of a still image (having a format such asJPEG or TIFF) as well as video from, for example, a digital television(DTV), a DVD, a CD, set top box (with satellite DSS or cable signal),the Internet (also commonly referred to as the WEB), and the like. Inthis way, a digital image source 204 can provide any number and type ofwell-known formats, such as, JPEG, BMP, TIFF, BNC composite, serialdigital, parallel digital. RGB, or consumer digital video. When thedigital image source 204 is a digital television, the input signal is adigital video signal having any number and type of well known digitalformats such as, SMPTE 274M-1995 (1920×1080 resolution, progressive orinterlaced scan), SMPTE 296M-1997 (1280×720 resolution, progressivescan), as well as standard 480 progressive scan video. Alternatively, asan analog image source 206, the analog signal provided includes analogsignals derived from, for example, an analog television, still camera,analog VCR, DVD player, camcorder, laser disk player, TV tuner, scanner,set top box (with satellite DSS or cable signal), game machines, imagingkiosks, as well as the WEB, and the like.

[0099] In order to convert the analog signal provided by the analogimage source 206 to an appropriate digital signal, an analog-to-digitalconverter (A/D) 208 coupled to the analog image source 206 converts ananalog voltage or current signal into a discrete series of digitallyencoded numbers (signal). In this way, a digital image data wordsuitable for digital processing is thereby generated. Any of a widevariety of A/D converters can be used, such as for example thosemanufactured by: Philips, Texas Instrument, Analog Devices, Brooktree,and others.

[0100] In the described embodiment, a selector unit 209 selects thedigital image stream from either the digital image source 204 or the A/Dconverter 208 to form an input digital image stream 210 to a digitalimage processing engine 212. Such an engine can be implemented asdescribed in more detail with reference to FIGS. 4-6. In some cases, theinput stream 210 has embedded therein digital image in the form of adigital negative also referred to as a reference. In other cases,however, the digital image embedded in the input stream 210 can take theform of a resultant proxy image 300 (i.e., modified image file) havingeither an embedded edit list file 302 as illustrated in FIG. 3A orhaving a pointer that identifies an external edit list file 304 as shownin FIG. 3B. In either case, the edit list files 302 and 304 provide apixel by pixel image processing instruction set operable oncorresponding pixels in the associated digital negative.

[0101] In the described embodiment, an external edit list source 213provides external edit list data to the digital image processing engine212. The external edit list source 213 can represent any number and typeof devices suitably arranged to provide appropriately configured editlist data used to modify, an associated digital image. Such devicesinclude, but are not limited to, user interface (UI) controllers (at thecontrol of an application or other device), a resultant proxy imagegenerator (such as a digital still camera having embedded imageprocessing capabilities), commands to a server computing device from aclient computing device, printers and other like peripheral devices thatgenerate system commands (for example, re-rasterizing a particulardigital image to suit a particular printing need), and other deviceswell known to those skilled in the art. In the preferred embodiment, anopen standard such as XML is desired. However, any other format, open orproprietary, can be used to represent the “edit list”.

[0102] The image processing engine 212 operates on the digital imageembedded in the incoming digital image stream 210 based, in part, uponthe accompanying edit list data, if any. For example, in the case wherethe digital image embedded in the input stream 210 is a digital negativetype image (i.e., a reference image), then the incoming digital negativegenerally has no associated edit list data (since it is by definition animage to be used as a reference). This is typically the case when theimage processing engine 212 detects that the incoming digital image hasno associated edit list data. In this case, the image processing engine212 treats that incoming digital image as a reference digital image andpasses it through to an output unit 216 without further processing.

[0103] When the image processing engine 212 detects that the incomingimage is a resultant proxy image (i.e., one that has been modified), theimage processing engine 212 then determines the method of association(i.e. whether the corresponding edit list data is embedded in theincoming image file as an edit list (as illustrated in FIG. 3A), if theimage file points to the appropriate edit list file (as illustrated inFIG. 3B), or has been generated by an external source (such as a UIcontrol, printer, or other imaging device)) and not actually associatedwith the resultant image in any way.

[0104] The edit list data associated with the resultant image isidentified and known to be of the specific format known by the imageprocessing engine 212. Of course, it is possible for a digital negativeor the resultant image to include other data as required by anapplication that could be ignored by the image processing engine 212 ifit is deemed irrelevant. This may be the case when file formats aredeveloped that embed other types of data in the edit list in itspreferred embodiment as an XML file format.

[0105] It is particularly advantageous that an edit list (commands forthe specific imaging operations) of operations that are applied to agiven photo are included in a particular edit list file can be storedwith the resultant image. This arrangement allows for those operationsidentified by the edit list to be reapplied to the digital negative, ifdesired. It also provides a way for the user to “undo” certainoperations or “disabled” or otherwise turn off certain operations thathave been applied to the photo. In so doing, this mechanism allows theseoperations to be reapplied at any resolution such that higher qualityresults can be achieved.

[0106] In some cases, however, the image-processing engine 212 detectsexternal edit list data only that is supplied by, for example, aprinter. In such cases, the image processing engine 212 is capable ofusing the external edit list data file to process any selected imagefile based upon the external edit list data. For example, if aparticular printing job requires a printed image to be enlarged from theoriginal, then the associated edit list can have re-rasterizinginstructions which have the effect of scaling the original image towhatever size is deemed to be appropriate. In the described embodiment,the image processing engine 212 processes the received digital imageaccording to the instructions (if any) included in any edit listassociated with (embedded in or pointed to by) the incoming image file.When the image processing engine 212 modifies an incoming digital imagebased upon acquired edit list data, the modified image can be either amodified resultant proxy image with an embedded edit list or it can be amodified resulting proxy image that points to an edit list file havingthe appropriate edit list data in the form of an edit list, for example.

[0107] In the case of a printer generated edit list and the incomingimage file is a digital negative, then the image processing engineperforms the image processing operations from the edit list (such asenlarging the image) on the incoming digital negative and then passes itto an output selector 214. In the cases where the incoming resultantimage file has its own edit list either embedded in or pointed to, thenthe image processing engine 212 performs the image processing operationsaccordingly on only those pixels in the incoming digital image soidentified by the corresponding edit list data in the edit listassociated with the resultant image as well as those editing commandsgenerated by the printer (such as enlarging the original). In both ofthese cases, a physical “resultant image” might not saved, but passedthrough to the printer port 220 for output.

[0108] The use of the externally generated edit list could also serveother purposes. For example, an application or device (such as aprinter) could allow the user to toggle viewing/output display states.For instance, suppose every displayed output image should have animaging operation applied (such as color adjustment needed for ade-saturation step before display on an NTSC television). These commandscould be use as input to the edit list source 213 and automaticallyapplied to all photos processed. In some situations, where the imagesare not saved, but printed (or just displayed), a physical resultantimage might not be persisted (beyond an intermediate image/file that issent to the printer or display).

[0109] In a similar way, the edit list source 213 could be driven by anapplication or device to allow for batch operations. For example, whenimages are first uploaded to a Web site, each photo could beautomatically processed (made better via AutoFix imaging operation) whenthey are uploaded. However, in this case, the original digital negativecould be preserved, but with a modified resultant image that contains anedit list of operations (such as AutoFix) with the link back to theoriginal digital negative.

[0110] In any case, all output image streams are output to the outputselector 214 and based upon user supplied selection criteria, one of theoutput image streams is selected to form an output image stream 215. Inthe described embodiment, the output image stream 215 forms an input toan output unit 216 that is coupled to external circuitry that caninclude an image display unit 218 capable of displaying the imageprovided by the output unit 216. In still other cases, the output unit216 can be coupled to an output port 220 capable of being coupled toother external circuitry (not shown) such as a computer that, in turn,provides an appropriate display apparatus. In the described embodiment,the output port 220 includes I/O ports such as a parallel port, a serialport, a system bus, a USB port, 1394, SCSI, a TV signal output port, aPCMCIA port, network port (to connect to the Internet), as well as amodem port. As such, the output port 220 can also be arranged totransmit the image provided by the output unit 216 (in a format such asraw RGB, JPEG, TIFF, PostScript, a proprietary image format, etc.) to aninterconnected network of computers (such as the Internet). In this way,any of the interconnected computers can be used to, for example, viewthe image embedded in the image stream 215 thereby allowing any numberof end-users to view the embedded image.

[0111] It is a particular advantage of the inventive digital imageprocessor system 200 that the image is processed (rendered) at whateverresolution is required by a particular output device (i.e. display,printer, etc.). The mechanism for accessing and rendering of an image tothe desired resolution is through an encoding mechanism that links thedigital negative and an edit-list of operations that have been appliedto the resultant proxy image. This provides the benefit for the user toview (and process) a much smaller proxy image file as necessary, butwith the ability to access the original digital negative and rasterizeit at any resolution. Furthermore, the proxy image (if it is stored in aJPEG format or an alternate format as desired) can serve both as acontainer that holds the reference to the digital negative and the editlist as well as a standard image file that can be viewed using aconventional application.

[0112]FIG. 2 illustrates a particular implementation of the imageprocessing engine 212 in accordance with an embodiment of the invention.As implemented, the image processing engine 212 includes an inputcontroller 250 coupled to an image processor 252 and an edit listprocessor 254. In operation, the input controller 250 determines whetheror not a received input signal is a resultant proxy image, a referenceimage, or an external edit list data file. In the case when the inputsignal is a reference image (digital negative) and no external edit listdata is supplied, the input controller 250 outputs the referencedirectly to the output unit 216 without further image processing.

[0113] In the case when the input signal is a reference image (digitalnegative) and there is external edit list data is supplied (such as froma printer), the input controller 250 passes the appropriate edit listdata to the edit list processor 254. When edit list data is supplied,the edit list processor 254 will ask the input controller to request thenecessary files (other referenced edit lists). Once the input controllerobtains the necessary files, they will be passed to the image processor252. The image processor 252 then uses the image processing operationsprovided by the edit list processor as defined by the embedded edit listdata to modify the original digital negative image to form a resultingproxy image with embedded edit list data in one case or in another casea resulting proxy image that points to an external edit list data fileprovided by the edit list processor 254.

[0114] On the other hand, if the input signal is a resultant proxy imagewith embedded edit list data, then the input controller 250 passes theappropriate edit list data to the edit list processor 254. When editlist data is supplied, the edit list processor 254 will determine fromthe edit list data how to access the original digital negative and askthe input controller to request the necessary files. Once the inputcontroller obtains the necessary image files, they will be passed to theimage processor 252. The image processor 252 then uses the imageprocessing operations provided by the edit list processor as defined bythe embedded edit list data to modify the original digital negativeimage to form a resulting proxy image with embedded edit list data inone case or in another case a resulting proxy image that points to anexternal edit list data file provided by the edit list processor 254.

[0115] When the edit list processor 254 must process several differentedits list (such as one referenced by the resultant image (or possiblymultiple ones recursively referenced) or from an external device (suchas a printer), the edit list processor will have to generate anaccumulated edit list that represents sum of all edit operations. In thepreferred embodiment, XML is used, however, other format are equallyvalid. If there are conflicts between the various edit lists, theapplication will need to reconcile these differences and process theedit list as is required.

[0116] In those cases where the resulting proxy image points to an editlist file or where the edit list data is in the form of an external editlist file, the image processing engine 212 uses the pointer to generatea reference (such as a URL) to locate and download the appropriate editlist file(s). The retrieved edit list data is then used by the edit listprocessor 252 to generate the resulting proxy image accordingly.

[0117] It should be noted that if the input controller 250 cannot accessthe original digital negative(s), other multi-media assets, or retrieveother parts as described in the edit list, the image processor 252 can,at the direction of the controlling application, directly output theoriginal inputted proxy image. In this way, the user will still see aversion of the output image, even though it has not been re-processed.

[0118] It should also be noted that if the image processor 252 cannotprocess the commands specified by the edit list processor 254 for whatever reason, the image processor 252 can, at the direction of thecontrolling application, directly output the original inputted proxyimage. In this way, the user will still see a version of the outputimage, even though it has not been re-processed.

[0119] It should also be noted that the edit list data associated with areference proxy image can point to multiple digital negatives and othermultimedia content used in the creation of the reference proxy image.This can include vector images (such as PostScript, EPS, or PDF), rasterimages (such as JPEG, TIFF, or PNG), text strings, video, sound, and thelike.

[0120] Referring now to FIG. 4 showing a distributed system 400 inaccordance with an embodiment of the invention. In the distributedsystem 400, the image processing engine 212 is coupled to a hostcomputer 402-1 by way of the output port 220. The host computer 402-1can be, in turn, coupled to a server computer 404 by way of a network406 or other host computers 402-2 through 402-n or an imaging appliance403-1 through 403-n. It is also possible that an imaging appliance 403-1(such as a digital still or digital video camera, set-top box, gamemachine, or photo appliance), which has embedded inside of it an imageprocessing engine 212, can be coupled to a server computer 404 by way ofa network 406 or other host computers 402-1 through 402-n or otherimaging appliances 403-2 through 403-n.

[0121] Further, it is also possible that the image-processing engine 212can be directly connected to the network 406, independent of computer402-1 through 402-n.

[0122] The network 406 can take the form of a distributed network ofinterconnected computers, such as can be found in the Internet, or itcan be a small scale intranet type system found in, for example, smallcompanies and the like. In the described embodiment, the server computer404 can include a server image processor unit 408 arranged to performhigh speed image processing operations based upon edit list data, forexample, found in an edit list associated with a particular image file.The server image processor unit 408 is arranged to perform those imageprocessing operations that are either too time consuming or resourceintensive to be efficiently performed on the host computer 402-1 or theimaging appliance 403-1.

[0123] One such case is when a high resolution rendered image with alloperations applied (with maximum quality imaging algorithms) is storedin the standard resultant image file (i.e. JPEG). Associated with thisJPEG file (either embedded or linked externally) is the edit list(commands for the specific imaging operations) and reference to theoriginal digital negative. Using this arrangement, a conventionallyconfigured application can ignore the edit list and digital negativereference and simply display the resultant image. It is one of theadvantages of this invention that allows an application or other devicethat does not implement this described system to still have the abilityto display the proxy resultant image (at whatever resolution it wassaved). However, the application or other device that does not implementthis described system can not take advantage of all the other advantagesof this invention.

[0124] In the described embodiment, the digital image processing engine212 can be included in any of a number of digital appliances 403-1 suchas for example, a digital still or digital video camera, set-top box,game machine, photo appliance, and the like. Using the example of adigital still camera, instead of storing the high-resolution resultantimage, an application may decide to store only the low-resolutionresultant image. For the case of a digital camera based application,this may be more desirable given the memory and processor constraints ofthe embedded system. For this case, the user can still choose the imageoperations as specified in the edit list, but only a low-resolutionresultant image is generated and stored locally, in addition to theoriginal digital negative

[0125] In some cases, such as when the digital image processing engine212 is part of a set-top box with limited or no local storage space, allcomponents, both digital negative and the low-resolution resultant imagewith its edit list could be stored remotely on the network.

[0126] This has several benefits. First, the amount of processing power(and internal memory requirements) of either the local computing deviceor the remote computing device (where ever the image processing occurs)is reduced since much fewer pixels are processed. Second, the size ofthe resultant image file is much smaller since only a low-resolution (orthumbnail) version of the image is persisted. Most importantly, theoriginal digital negative is preserved, along with the smaller resultantimage. This is critical since digital camera storage is veryconstrained. In this case, the original digital negative will be savedand a lower-resolution (and much smaller) resultant image with itsassociated edit list will be saved to the digital still camera.

[0127] Once the image has been processed, the low-resolution resultantimage can be sent over the network 406 to any number of other hostcomputers 402-1 through 402-n or imaging appliances 403-1 though 403-nfor viewing on their respective displays.

[0128] This has the advantage of only consuming a limited amount ofbandwidth for the transfer of the lower-resolution thumbnail imagethrough a low-bandwidth connection. However, at the request of theapplication, the original digital negative could be accessed such thatthe edit operations could be reapplied to generate a higher-resolutionresultant proxy image.

[0129] Depending on the capabilities of the host computer or the imagingappliance, this processing could either occur on the host computer 402-1or imaging appliance 403-1, or on one of the system attached to the hostcomputer via the network 406, such as another the server image processorunit 408 that is part of the server 404.

[0130] It should be noted that for either case described above, the filethat contains the edit-list could take on several forms. It may bedesirable to generate a JPEG file that contains the resultant image andembeds the edit-list and the reference to the digital negative. This hasthe benefit that the user has one encapsulated file to maintain. Anotherapproach involves creation of an edit list file that contains theedit-list, the reference to the digital negative, and a reference to theresultant image (possibly a JPEG file that may or may not includemetadata about the edit-list and/or reference to digital negative). Athird approach would involve the complete encapsulation of not only theresultant image and the edit list, but also a digital negative. Thiscould take the form of an image file (JPEG, TIFF, etc.) or a general“collection” file (such as a compressed ZIP or JAR (Java Archive)). Thismay be desirable as a distribution mechanism, but maintains theproperties described by this invention. It should also be noted thatwith any of these variants, multiple resolutions may also be stored(embedded or linked) at the application's discretion.

[0131] With conventional Web editing techniques, if a user is editing aphoto that resides on the Web, the entire image must be downloaded andprocessed locally. However, this invention allows for a more efficientalternative approach that calls for a low resolution-image to bedownloaded from the remote computing device (such as the server computer406) and processed by the local computer device (such as the hostcomputer 402-1 or photo appliance 403-1). The higher resolution imagedata would only be required by local computing device and downloadedfrom the remote computing device if needed. For example, referring toFIG. 4, a user working on a local computing device (such as a photoappliance 403-1 without access to local storage) can efficiently savechanges to the server computer 404 over the network 406 even when thenetwork 406 is a low bandwidth network such as a modem. This results inmuch less bandwidth, assuming only a low-resolution resultant image istransmitted with the edit list and other metadata to the remotecomputing device (such as the server computer 406). Alternatively, theuser may choose to save the resultant image locally with the edit list,in for example, the host computer 402-1 with a link to the originaldigital negative that can be reference at any node in the network 406.

[0132] Another advantage of the distributed nature of this invention isthat the actual image operations can be processed either locally in anyof the local computing devices (such as the host computer 402-1 or photoappliance 403-1) or a remote computing device (such as the servercompute 404), as required by the application. It also is possible toaccommodate a mixed mode model in which the low-resolution image isprocessed locally (performing the commands in the edit list), but ahigh-resolution image is processed using the same edit list remotely onthe server computer 404.

[0133] The invention will now be described in terms of a digitalappliance, such as a digital camera having relatively small LCD screensfor display and viewing. Since, the LCD screens are very small and onlyrequire a low-resolution image, processing of the image is onlyperformed on a thumbnail or low-resolution image and the digitalnegative will be saved to the camera's memory card.

[0134] Referring to FIG. 5, a digital camera system 500 in accordancewith an embodiment of the invention is described. In this example, thecamera system 500 is capable of color correcting digital images eitheron the fly or after the fact. In some cases, it may be advantageous togenerate an edit list describing the color correction, such as an ICCprofile, that is required in conjunction with an uncorrected referenceimage, thus preserving the original digital negative. Also stored willbe a resultant image, most likely of a low-resolution thumbnail imagewith the color correction applied such that it can be displayed on theLCD of the camera. The uncorrected reference image (digital negative)along with the edit list can then be downloaded to either a local hostcomputing device or can be sent to a remote computing device for furtherimage processing. Alternatively, if transmission of the digital negativeis not possible, the resultant proxy image, along with the edit list,can be transmitted and displayed on local host computing device orremote computing device. At a later time, the edit list, which isassociated with the resultant proxy image can be processed and appliedto the digital negative, which is referenced from the edit list.

[0135] Typically the digital camera system 500 includes the image source202 that includes an optical lens 502 for focusing light rays onto aimaging capture unit 504. The imaging capture unit 504 typically reliesupon an array of light sensitive photo optic cells 506 capable ofconverting the light rays, in the form of photons, received from theoptical lens 502 into representative analog signals. Most commonly, thephoto optic cells take the form of charge coupled devices (CCDs),although other devices such as CMOS receptors may be used as w ell.

[0136] As is well understood in the art, each CCD array 506 hasassociated with it a specific color filter array (CFA) 508. In mostapplications, the CFA 508 is an empirically derived pattern ofindividual color filters each associated with a specific CCD cell in theCCD array. When the analog signals representative of the captured imageare produced by the image capture unit 504, they are sent first to ananalog to the digital (A/D) converter unit 208 which converts therepresentative analog signals into digital signals representative of theimage. The digital signals are then passed to a digital signal processor(DSP) 518 where they are converted to an appropriate digital format.

[0137] In some embodiments of the invention, an image compressioncircuit 520 as well as a memory 522 both receive the appropriatelyformatted digital signals. The image compression circuit 520 operates todigitally compress the received digital images in order to reduce theamount of resources required to further process the digital signals. Onesuch formatting scheme referred to as JPEG is commonly used, althoughthere are a wide variety of suitable picture formats.

[0138] In one embodiment, the digital image stored in the memory 522 canbe sent by way of a memory interface (not shown) to a memory slot 524capable of receiving a memory card suitable for storing the processeddigital signals. Such memory cards include “floppy” disks, flash EPROMcards, R/W compact disc (CD), SmartMedia, CompactFlash, Sony'sMemoryStick, DVD, DAT, and the like. Alternatively, the digital imagestored in the memory 522 can be sent to external circuitry, such as acomputer, by way of the output port 220 for viewing or, in some cases,additional image processing. In the situation where a number of imagesare stored in the memory 522, a user can pre-select any number of imagesfor additional processing by viewing the images (generally in athumbnail form) using a viewing device such as a viewfinder 526.

[0139] In the case where the digital image is to be processed by anexternal computer, for example, an edit list generator 523 generates anedit list consistent with those color corrections required to compensatefor the ambient light conditions. The edit list generator 523 can eitherembed the edit list in the reference image or it can create a separateedit list which is associated with the reference digital image, by forexample, a pointer.

[0140] If it is determined that the digital image is not to be processedby the image processing unit 212, then the image stored in the memory522 can be directly sent to either the display unit 526, to a devicecoupled to the memory slot 524, or to external circuitry, such as acomputer, by way of the output port 220. In this case, the externalcomputer can actually perform the color correction as indicated by theedit list generated by the edit list generator 523.

[0141] In the case where the output device includes a printer, the useris able to print out the photos captured with the digital camera 500. Ifthe user is printing out a digital negative, those photos can bedirectly printed as specified by the user. However, if the user printsout photos that have been modified that contain only a low-resolutionthumbnail, it may not be at a high enough resolution to meet the needsof the printer. For this scenario, the printer can re-rasterize theimage by applying the edit list to the original digital negative toobtain high quality output results as desired.

[0142] It should be noted that the invention also provides for a veryefficient means of describing a print job that allows rasterization tooccur at the printer. Typically, an application (or print driver)rasterizes an image (or composition of images) to the specific printresolution locally on the client, that is then output to the printer.While this approach is acceptable for local, client-side printing, it isclearly inefficient for server-side remote printing.

[0143] An alternative approach involves sending the edit list andreferences to the digital negative(s) (the contents associated with theresultant image), to the server for processing. These operations canthen be reapplied to the digital negative to produce the desired printresolution on the server. This approach clearly preserves bandwidthbecause the fully-rasterized image is not sent to the print server. Thisis particularly important if the print job involves any vector-basedartwork that contains a highly compressed description of a page layout.

[0144] Additionally, the invention is well suited for printing of“package prints.” As well known in the art, package prints are those inwhich one photo is printed on a variety of different print layouttemplates (i.e. one 5×7 and two 3×4). If only one photo is printed, butto several different orientations and sizes, only one copy of the photomust be transferred. Equally important, it reduces the amount ofprocessing power required on the client side since the rasterization canbe off loaded to the server.

[0145] Referring now to FIG. 6, a flowchart is shown detailing a process600 processing a digital image by the digital image-processing engine inaccordance with an embodiment of the invention. The process 600 beginsat 602 when a particular digital image is opened by the image-processingengine. At 603, a determination is made if the image should be directlydisplayed, without any further processing, in which case control ispassed to 630 where the image is passed to the output device for display(or print or to a file). This could occur in several circumstances. Ifthe application does not have knowledge of this invention, it willsimply display the image. If the application does have knowledge of thisinvention, but wants to quickly display the contents of the image,whether it is a resultant proxy image or the digital negative (an imagewithout any associated edit list), it can bypass any additionalprocessing and simply display the image.

[0146] If the image should not be directly displayed, then at 604 and606, a determination is made if there is an edit list data embedded inor if external edit list data is referenced to, respectively, the openedimage file. If it is determined that there is no embedded or referencededit list data with the associated digital image, then control is passedto 630 where the image is passed to the output device for display (orprint or to a file).

[0147] If it was determined that there was either embedded or referencededit list data associated with the input digital image, then at 610 adetermination is made to what the desired output resolution should be.Once this is known, then at 612 the first edit list element is processedand any necessary digital negatives are retrieved at the resolutionappropriate based on the value determined in 610. At 614 a determinationis made to see if additional edit list elements or more digital negativeelements are to be retried, and if so, control is passed back to 612.Otherwise, control is pass to 616 where the edit list elements anddigital negative(s) are processed, based on the desired resolutiondetermined in 610, resulting in a new output image.

[0148] At 618, a determination is made whether or not the edit list dataassociated with the resultant output image generated at 616 should alsobe generated. In many cases, this will simply be the original edit listprocessed at 612. If no edit list data is to be generated, the imagegenerated from 616 is passed to 630 where the image is passed to theoutput device for display (or print or to a file).

[0149] If an edit list is to be generated, a determination is made at620 whether it should be embedded with the image file or external as aseparate file. If the edit list data is determined to be embedded, thenthe edit list data is embedded in the resultant proxy image at 622,otherwise the edit list data is stored in an associated edit list file,such as, for example, an edit list at 624. Regardless, control is thenpass to 630 where the image is passed to the output device for display(or print or to a file).

[0150] It should be noted, if the resolution generated (and requested at610) does not exactly match the output device, it may be necessary toresample the output image to match the output device. While the mostaccurate results would be achieved if the image were re-rasterized,based on the edit list and digital negative(s), this may not always bedone. For example, when the user zooms in/out to/from an image, it maybe desirable to quickly resample the image to get the desired resolutionas needed.

[0151] In a particular scenario, a user creates an intelligent “e-card”as detailed by flowchart of a process 700 as illustrated in FIG. 7 inaccordance with an embodiment of the invention. The process 700 can becarried out in any suitably arranged digital appliance, such as forexample, the digital imaging set-top device 500 described above withreference to FIG. 4 in conjunction with the distributed network 400.Therefore, a user chooses a postcard, for example, at 702 while choosingphoto image(s) at 704 to be included in the postcard. Once selected, theuser positions the photo images inside the selected postcard at 706,which may involve zooming and panning the photo inside the composition.Once satisfied, the composition is saved as a single image at 708 at alow resolution. In addition at 708, the edit list will also be saved,either embedded inside or associated with, the generated resultant proxyimage. Once saved, the user can email the image to the receiver(s) at710. Control is then passed to 602 when the image can be opened

[0152]FIG. 8 illustrates a process 800 for facilitating efficient videoediting in a distributed environment in accordance with an embodiment ofthe invention. At 802, a user drops off a videotape (in digital oranalog format) to a video processor or merchant that supports thisprocess. At 804, a determination is made if the videotape is in analogformat or digital format. If in analog format, the tape is convertedinto digital format at 806. In either case, control is then pass to 808where a low-resolution digital video stream is created as well asdetermining at 810 whether a higher resolution video stream is to becreated If a higher resolution video stream is to be created, then thehigher resolution video stream is created at 812, otherwise, the lowresolution video stream is either placed on a CD, DVD, or other digitalmedia (or uploaded to the network where it can be accessed) at 814. At816, the user accesses the low-resolution digital video stream andperforms the necessary edits (most likely from an video application orconsumer electronics device at home or a kiosk in a store).

[0153] At 818, these edits are stored an edit list in an appropriateformat (such as XML or other compact open format) such that they can beprovided to a video processor. At 820, the video processor accesses theedit list generated in 818, either through the network or someread-write media provided by the customer. At 822, a determination ismade whether or not a higher resolution video stream is available (i.e.made available at 812) or is required at a different resolution than iscurrently available. If a higher resolution video stream is notcurrently available or a different resolution is required, then thehigher resolution or different resolution video stream is created at824. In any case, at 826, the higher resolution video stream isretrieved and at 828, the edit list is applied to a higher-resolutionvideo stream (i.e., rasterization of video stream based upon the editlist at the appropriate resolution). At 830, the merchant outputs theresults from the rasterization of the edit list at the desiredresolution to an appropriate output medium, such as DVD, CD, digital oranalog tape, so the user can view the edits they had performed on thelow-resolution video stream, but rendered at a much higher resolutionoutput.

[0154]FIG. 9 illustrates a computer system 900 or any imaging-enabledcomputing appliance/device that employed to implement the invention. Thecomputer system 900 or, more specifically, CPUs 902, may be arranged tosupport a virtual machine, as will be appreciated by those skilled inthe art. As is well known in the art, ROM acts to transfer data andinstructions uni-directionally to the CPUs 902, while RAM is usedtypically to transfer data and instructions in a bi-directional manner.CPUs 902 may generally include any number of processors. Both primarystorage devices 904, 906 may include any suitable computer-readablemedia. A secondary storage medium 908, which is typically a mass memorydevice, is also coupled bi-directionally to CPUs 902 and providesadditional data storage capacity. The mass memory device 908 is acomputer-readable medium that may be used to store programs includingcomputer code, data, and the like. Typically, mass memory device 908 isa storage medium such as a hard disk or a tape which generally slowerthan primary storage devices 904, 906. Mass memory storage device 908may take the form of a magnetic or paper tape reader or some otherwell-known device. It will be appreciated that the information retainedwithin the mass memory device 908, may, in appropriate cases, beincorporated in standard fashion as part of RAM 906 as virtual memory. Aspecific primary storage device 904 such as a CD-ROM may also pass datauni-directionally to the CPUs 902.

[0155] CPUs 902 are also coupled to one or more input/output devices 910that may include, but are not limited to, devices such as videomonitors, track balls, mice, scanners, PC-card readers, digital cameras,digital video, remote control units for TVs, CRs, DVD players, and thelike, keyboards, microphones, touch-sensitive displays, transducer cardreaders, magnetic or paper tape readers, tablets, styluses. voice orhandwriting recognizers, or other well-known input devices such as, ofcourse, other computers. Finally, CPUs 902 optionally may be coupled toa computer or telecommunications network, e.g., an Internet network oran Intranet network, using a network connection as shown generally at912. With such a network connection, it is contemplated that the CPUs902 might receive information from the network, or might outputinformation to the network in the course of performing theabove-described method steps. Such information, which is oftenrepresented as a sequence of instructions to be executed using CPUs 902,may be received from and outputted to the network, for example, in theform of a computer data signal embodied in a carrier wave. Theabove-described devices and materials will be familiar to those of skillin the computer hardware and software arts.

[0156] While the present invention has been described as being used witha digital image or video system, it should be appreciated that thepresent invention may generally be implemented on any suitable digitalimage system. Additionally, this invention is not limited to digitalimages, but extends to any image type where a high-resolution dataobject representation can be approximated by a lower-resolution proxyimage, including-video imaging application and systems as well as audiosystems.

[0157] It should be noted that the digital negative itself could be astandard image file type (JPEG, TIFF, PNG, etc.) or a propriety imageformat. Alternatively, the digital negative could be a recursiveresultant image that points to multiple digital negative(s) and itsassociated edit list. This provides a powerful mechanism in which alow-resolution proxy image could be subsequently edited and/or placedinside a print job, package print, card, or digital album. Then anapplication, or server application, that supports this invention will beable to reapply all the editing operations recursively to generate thehighest quality output image as required by the output device. For thoseapplications that do not support this invention, they will be able toprint the top most resultant image that is the composition of allrecursively added inner images.

[0158] The resultant proxy image may also be a standard image file type(JPEG, TIFF, PNG, etc.) or a proprietary image format, with a resolutionspecified by the application that creates the resultant proxy image. Itis also possible in some cases that a resultant proxy image does noteven exist (or is “empty” with out a physical pixel representation). Asdescribed above, the edit list of operations may describe a print layouttemplate with the associated digital negatives (and possible edit listof editing operations) and other multimedia assets. In this case, thecreation of the resultant image is not needed since the printer servicewill process the edit list and its associated digital negatives andother assets and generate the desired output. As a convenience, thiscould be represented as a blank JPEG file with an embedded or externallylinked edit list.

[0159] It should be also noted that the edit list could be persistedthrough many different mechanisms. The preferred embodiment uses XML(eXtensible Markup Language), but other formats, both standard andproprietary could be used to implement this invention. Further, thisedit list can either be included with the resultant image or linked toit via a separate file.

[0160] The distributed nature as described in the recursive exampleabove also provides a powerful and flexible workflow such that separatedesigners can work on different parts of this recursive structurewithout effects to the others.

[0161] Therefore, the present examples are to be considered asillustrative and not restrictive, and the invention is not to be limitedto the details given herein, but may be modified within the scope of theappended claims along with their full scope of equivalents.

What is claimed is:
 1. A method of processing a stored videostream,comprising: (a) determining if the stored videostream is a digitalvideostream; (b) creating a low resolution digital videostream basedupon the stored digital videostrean; (c) storing the low resolutiondigital videostream at a digital storage medium; (d) editting the storedlow resolution digital videostream; (e) creating a resultant digitalvideostream and an associated edit list based upon the editing; (f)accessing the edit list by a video processor; (g) rendering a highresolution videostream by the video processor based upon the edit list;and (h) storing the rendered high resolution videostream on a selectedstorage medium.
 2. A method as recited in claim 1, further comprising:if it is determined that the stored digital videostream is an analogvideostream, (i) converting the stored analog videostream to the digitalvideostream.
 3. A method as recited in claim 1, wherein the digitalstorage medium is selected from a group comprising: a DVD, CD, a digitalvideo tape, a flash memory device, a hard drive.
 4. A method as recitedin claim 2, wherein the operations (a)-(c) and (i) are performed at afirst node.
 5. A method as recited in claim 4, wherein the lowresolution videostream is transferred from the first node to a secondnode coupled thereto.
 6. A method s recited in claim 5, wherein theoperations (d) and (e) are performed at the second node.
 7. A method asrecited in claim 5, further comprising: (j) transferring the edit listto the first node.
 8. A method as recited in claim 7, wherein theoperations (f)-(h) are performed at the first node.
 9. A method asrecited in claim 1, wherein the selected storage medium is selected froma group comprising: a DVD, CD, a digital video tape, a flash memorydevice, a hard drive, analog video tape, VCR tape.
 10. A system forprocessing a stored videostream, comprising: a first means fordetermining if the stored video is a digital videostream; a second meanscoupled to the first means for creating a low resolution digitalvideostream based upon the stored digital videostream; a third meanscoupled to the second means for storing the low resolution digitalvideostream at a digital storage medium; a fourth means coupled to thethird means for editting the stored low resolution digital videostrean;a fifth means coupled to the fourth means for creating a resultantdigital videostream and an associated edit list based upon the editing;a sixth means coupled to the fifth means for accessing the edit list bya video processor; a seventh means coupled to the sixth means forrendering a high resolution videostream by the video processor basedupon the edit list; and an eighth means coupled to the seventh means forstoring the rendered high resolution videostream on a selected storagemedium.
 11. A system as recited in claim 10, wherein the first, secondand third means are directly connected to a first node.
 12. A system asrecited in claim 11, further comprising a means for transferring the lowresolution videostream from the first node to a second node coupledthereto.
 13. A system as recited in claim 12, wherein the fourth andfifth means are connected to the second node.
 14. A system as recited inclaim 13, further comprising a means for transferring the edit list fromthe second node to the first node.
 15. A system as recited in claim 14,wherein the sixth, seventh, and eighth means are connected to the firstnode.
 16. A system as recited in claim 10, wherein the storage medium isselected from a group comprising: a DVD, CD, a digital video tape, aflash memory device, a hard drive, analog video tape, VCR tape.